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Abstract 

In this paper we derive the equations for Loop Corrected Behef Propagation on a 
continuous variable Gaussian model. Using the exactness of the averages for belief prop- 
agation for Gaussian models, a different way of obtaining the covariances is found, based 
on Belief Propagation on cavity graphs. We discuss the relation of this loop correction 
algorithm to Expectation Propagation algorithms for the case in which the model is no 
longer Gaussian, but slightly perturbed by nonlinear terms. 

1 Introduction 

Message passing techniques in graphical models allow for the computation of (approximate) 
marginal probabilities in a time interval scaling polynomially in the model size. Their discovery 
has consequently revolutionized several fields of applications in the past years, of which error 
correcting codes and vision are probably the most prominent examples. In many cases, the 
corresponding graphs are loopy, implying either that the error resulting from the application 
of loopy belief propagation (BP) is negligible for the particular model, or it can be tolerated 
for the particular purpose BP serves. In other cases more sophisticated refinements of BP are 
necessary, taking into account (part of) the loop errors. 

Finding the optimal treatment of these "loop errors" motivates an active field of research, 
in which different solutions applying to different model classes are developed. For models 
involving many short loops, like on regular lattices, CVM type approaches work well [2], or 
tree EP approaches [3j- The latter may also be applied to correct for an incidental large loop. 
Unifying frameworks like the Region graphs of [4] lead to general strategies for selecting the 
basic clusters underlying such approaches for general model classes. 
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A recent analysis has shown that the local update equations of BP may be interpreted as 
the zero order term of an expansion in "cavity connected correlations" . These quantities are 
parameterizations of the "cavity distributions", i.e., the distribution over neighbor variables 
of a central variable which has been removed from the graph. The Bethe approximation 
and BP are recovered when this cavity distribution is assumed to factorize, whereas the first 
order correction to the local update equations is obtained when one takes into account the 
pair cumulants [5]. Estimation of these pair cumulants is possible with extra runs of BP, 
allowing for new polynomial time algorithms, reducing errors to order when applying 

algorithms of which running time scales with an extra factor of [6j . Although this scaling 
seems heavy, the large benefit of the approach is that it does not require selection of basic 
clusters or underlying tree-structures, since it takes into account the effect of all loops that 
contribute to nontrivial correlations in the cavity distribution at once. 

The above "loop correction" strategy is applicable in the class of models where a per- 
turbative expansion around the Bethe approximation makes sense, i.e., in models with large 
loops and relatively weak interactions. The principal requirement is that the magnitude of 
pair variable cumulants of cavity distributions is an order smaller than the single variable 
cumulants, and third order cumulants are even smaller, etc. However, heuristics based on the 
strategy allow for other good algorithms performing well outside these parameter regimes [7]. 

So far the approach has been developed for discrete variable models on a more abstract 
O E] versus practical level [7]. In this paper we apply the idea to graphical models for 
continuous variables. We derive the loop corrected belief propagation equations for simple 
tractable Gaussian models, yielding a message passing scheme that, besides the correct av- 
erage marginals, also yields the correct variances. Besides that we discuss some approaches 
potentially applicable to cases in which extra function approximations are necessary, and the 
relation with expectation propagation. A by-product of our loop corrected belief propagation 
equations is an algorithm that calculates exact covariance matrices for Gaussian models like 
the one discussed in [Ij, but without explicitly using linear response. 

2 General ideas 

The error in the result of message passing techniques that are based on local approxima- 
tions for variables that interact on a graph, like belief propagation, may be viewed from two 
perspectives: 

• The error of the Bethe-approximation is due to the fact that loops in the graph are 
neglected, such that nontrivial correlations between two neighbors of one variable are 
neglected. 

• The error is due to the fact that the functional parameterization of the local marginals 
is not rich enough, such that it can at most be an approximation. 
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These two viewpoints may be argued to have the same meaning in the end, but nevertheless 
may lead to different strategies in the optimization of the approximation, or the improvement 
of the results. If the second viewpoint is the starting point for algorithms like expectation 
propagation, the first may be seen as the basic view for loop correction strategies. 

Since expectation propagation applies well to continuous variable cases, but loop correction 
schemes in the sense of [5l[71[6] have not been applied to continuous variable cases, it might be 
instructive to derive corresponding equations and compare them to expectation propagation 
approaches. 

With this motivation in mind, we will firstly analyze a loop correction scheme to BP in 
Gaussian models. Given this scheme, we will discuss possible generalizations suitable for cases 
in which the model is no longer tractable. 



2.1 Model introduction 



The model which we will initially consider is a Gaussian model of interacting variables, 
denoted by cr G IR'"^ of which the total probability distribution is given by 



P{a) 



N N 
1=1 j<k 



exp 



1 



2s, 

exp [Jjfc<7j(Tfe] 



(1) 



thus the variables i have their own Gaussian local potential with average fii and variance 
Si, but interact in a pairwise manner with variables j via the interaction J^. Obviously, 



Z = f da YiiLi i^ii'^i) Y[j<k ''Pjki'^jT'^k)- We will denote the neighborhood of variable i on the 
graph by i.e. di 



2.2 The "cavity equations" 

The following analysis will be based on the loop correction equations of which were ap- 
plied to discrete binary variables. The current generalization to continuous variables is a 
straightforward application of these ideas. We write down an expression for the joint prob- 
ability P^'^\ai,aj) of variables ai and aj on the model from which the interaction Jij has 
been removed in two different ways. The first is in terms of the cavity distribution of variable 
i, P^'^\aQ^), i.e., the joint distribution over the neighbors of i in the model from which i has 
been removed, and the second in terms of the cavity distribution of variable j, P^^\aQ.): 



piij) ( 



ai,aj 



(crgjexp 



-{(Ti - llif /{2Si) + Gi ^ JuCJl 
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With respect to this marginal distribution two ways of writing the moment 



(4) 



are 



which may be written in terms of effective measures 



l(^di\j 
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l(idj\i 
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where -^i^j and -^j^j are the corresponding normahzation constants. In terms of these mea- 
sures, the equations ([5]) lead to 



(o-i)j^j = m + Si ^ Ju{cri)i- 
ledi\j 



(7) 



The above procedure may be repeated for all other moments of the distribution P^^^^ai), e.g. 

{<Tf)j^i = Si + 14 + 2siiii Jii{ai)i^j + sf ^ JuJik{criak)i^j (8) 

ledi\j lMdi\j 



etc. The moments of the true marginal distributions are integrals with respect to different 
measures, e.g.: 

{(Ti) = + Si'^Jil{ai)i (9) 



ledi 



with (/(cr)) = / d(TP{(T)f{(T) and 




All these measures reduce to functions of the above mentioned cavity distributions, which are 
the unknown functions of interest. It is clear however, that so far we have not specified enough 
local equations to solve for the full cavity distributions P^'^^ag.). If we restrict ourselves, 
for the moment, to Gaussian models, we will be able to perform the integrations and find 
exact local message passing equations. We note that for a more general type of model, such 
local computations will be insufficient, but may be used as a basis for an approximation 
when an appropriate set of approximating functions is chosen, characterized by a finite set of 
parameters. 



3 Gaussian cavity distributions 

Notice that the belief propagation is recovered when one chooses to approximate the cavity 
distribution by a factorizing one, i.e. P^^\(tq.) ~ Ylj^OiQ^^^ ('^i)- This parameterization 
includes the exact result when the graph is a tree, since then there can be no nontrivial 
correlations between variables in any cavity set di when i is absent. When there are loops 
in the graph, corrections to this parameterization are desirable. Various parameterizations of 
these corrections are possible in principle, and in [5] it was suggested to expand the cavity 
distributions in the cumulants, an expansion that is appropriate when either interactions are 
weak or loops are long. 

For a Gaussian model, the cavity distributions are completely specified by their averages 
and covariances, such that including the second order cumulants (the first order correction to 
belief propagation) yields exact equations. In the following we investigate the structure of the 
corresponding equations and identify the exact correction to Gaussian belief propagation. An 
appropriate (and exact) parameterization of the cavity distribution is 



a-Qi) ~ exp 



(11) 



where we have decomposed the covariance matrix in a diagonal part (Dj) and an off-diagonal 
part (Ai), both having the dimensions of the cavity set. The Bethe approximation, for which 
cavity distributions factorize, corresponds to neglecting the off-diagonal components Ai. The 
matrices Di and vectors m* are found through consistency equations. In the following we will 
denote the vector Jj (again the dimensions of vectors m* and Jj are equal to that of the cavity 
set, \di\) for which Jij = as Jj. The consistency equations ([7]), by Gaussian integration, are 
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found to be 

{[{Dj + A,r^ - s.Jpfr'iiD, + A,)-^in^ + f^jj]]}^ 
= + s.jf {[(A + Ai)-^ - M^jf + Ai)-^m' + /iiJ^]} (12) 

From the relations of the variances, equation ([8|), we find: 

{[{D,+A^r'-s,i]if]-'}u = s,+sf3fm+Ar'-s^JiJ^r'J^ (13) 

For each cavity distribution Dj and m-' the number of pairs of equations is equal to the 
number of variables in the cavity set. Thus, given a covariance matrix A, the diagonals D 
can be determined with the second equation, and subsequently the average values m can be 
determined with the first equation. The marginal distributions then follow directly, since all 
variables are now known. Substituting (jlip into ([9]), we find 

(cj,) = n^ + SiJi [(A + Ai)-^ - SiJiJj] [(A + Ai)-'^m' + fiiJi] (14) 

and for the second moment 

{af) = {aif + Si{l + SiJl[{Di + Ai)-^-SiJiJf]-^J^} (15) 

The only obstacle in solving these exact equations is yet obtaining the off-diagonal covariances 
Ai for each cavity set di. 

Simply neglecting them, setting Ai = 0, we should recover the BP equations for the 
Gaussian model. 

Using response propagation it is possible to estimate the covariances, which leads to an 
improvement in the results when they are small for the binary case [SI [6] . In the Gaussian 
case, where results from response propagation are exact, [HIH], this procedure should thus 
yield exact results provided response propagation and belief propagation both converge. 



4 Loop corrected belief propagation 

Using the identity 

[A + XBX^]-^ = A-^ - A-^X{B-^ + X^A-'^Xy^X^A-'^ (16) 

we may write 

[(A + ^.)--..J^jf]- = A + A.+ (^; + ^-)y(^- + f (17) 
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Defining 



jf(A + ^.)Jl 



ai = 3f{Di + Ai)Ji 
e] = [[Dj+A,)i% = [A^J 



(18) 
(19) 
(20) 



and writing i;^ for the (diagonal) entries of Di where k runs over di, we find that equation 
(fT3l) yields 



1 — Si a - 



(21) 



After similar simplification of equation (I12p . the updates for the message variances and aver- 
ages become 



1 - Sitti J- - SjOj 



Si 



1 - Siuj 

and the final marginals are given by 



l€di\j 



1 - Sja) 



+ Jjimj 



l€dj\i 



(22) 
(23) 



Si 



1 - SiOi 



nii 



l€di 



(24) 
(25) 



Indeed the BP equations follow for Ai = 0, since in that case ej = 0, aj = X^^gg^^j Jf^vl. and 
ai = J2j£d Jij"^]^ such that the equations (modulo a transformation) reduce to the ones in 

The above equations allow one to explicitly interpret the meaning of the belief propagation 
messages, and write down expressions for their error. Indeed the messages in equation (j23|) 
represent averages and variances of cavity distributions, i.e., of the model in absence of a 
variable. An interesting side result in this respect comes from the observation in [S] that 
the averages calculated via belief propagation are exact when the algorithm converges. It 
follows that the message m-f calculated via equation (I23p is equal to mi on a graph from which 
variable j is removed calculated via ordinary belief propagation. In the next section we use 
this observation, together with similar arguments, to obtain some more exact results from 
ordinary belief propagation variables alone. 
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5 An alternative way to calculate the error in Vi for Gaussian 
models 



The form of the loop corrected behef propagation equations imposes a relation between the 
BP errors in Vi and the messages for Gaussian models. Comparing the result of equations 
(|25p with and without cavity covariances, one may show that 



m: 



LC 



a,- 



,BP 



led. 



i LC 



Now, since the BP averages are exact whenever BP converges [8], 



BPr^LC 



a,- 



RPi 



ledi 



i LC 



(26) 



(27) 



Due to the interpretation of the message averages as cavity parameters, we have access to 



?■ T C 

m , smce 



m 



(i) BP 
I 



(28) 



i.e. this is the average of variable / on the graph without i, which may be obtained by running 
BP on the graph without variable i. Thus by running BP on the original graph once and 
running it on the graph without i, we can calculate v^^'~' by using equation (I25p and writing 

(29) 



1 



afP + (mfP) 



(i) BP 



m 



i BP 
I 



provided that mf^ ^ 0. Similar considerations (see appendix) lead to a procedure for calcu- 
lating the entire covariance matrix using BP: condensing notation 



J- v^^ 



\m 



(OBP 



m. 



BPl 



BP 



Vi = 



m 



(i)BP 



BP i 



1- Si 

,BP 



(i) BP 



m 



i BP 
I 



rrii = 

we have the following equations: 



n*n^ + ViK'jKl + (fTjCTfc) 



(30) 

(31) 
(32) 

(33) 

(34) 
(35) 
(36) 
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These equations suggest inverting matrices by calculating correlation matrices on growing 
graphs might be a useful application. By subsequently attaching new variables to the graph 
and running BP, one finds the full correlation matrix with runs of BP, just as with the 
procedure described in [Ij, but the cost of the BP runs is halved since the graph is growing along 
with the BP runs. However, we should not overlook the fact that the equations above introduce 
large number of additions and multiplications, such that in the end the total computational 
complexity for inverting a sparse matrix is similar to other well-known methods. 

6 Nonlinear models: connections with EP 

The fact that loop corrections in the above form are able to correct for the total BP error in 
the (co)variances is of course due to the Gaussian nature of the model. In discrete models, 
exact parameterizations of the full distribution by use of local marginals only is in general 
not possible, but loop corrections are able to increase the accuracy of the Bethe approxima- 
tion. Thus the above formalism might seem a promising basis for extensions to models that 
are not exactly tractable, possibly as an alternative for related algorithms like Expectation 
Propagation (EP) [9j. Since BP may be viewed as a special case of EP, we may hope for some 
generalizations of loop corrections equations, with some relation to EP, applicable in cases 
where function approximations become necessary. However, the specific form of EP equations 
very much depends on the choice of the approximating family of functions one chooses. The 
equivalence with BP corresponds to a family of approximate EP functions that fully factorizes 
over the variables of the model [9]. For loop corrected BP strategies, we expect a relationship 
with EP approaches based on larger local neighborhoods. 

We will investigate the relation to EP by deriving equations for models with general non- 
linear single-variable potentials, i.e., ipi{(Ti) 'il^i{ai)e~^^^'^^\ as one might expect in vision 
problems with nonlinear observation functions. 

6.1 Full Gaussian EP 

The Gaussian loop corrections approach seems rather similar to an EP approach where one 
includes a full Gaussian in the approximating target distribution. The standard EP formalism 
for this approach is to choose as an approximate distribution 




(37) 



where 




(38) 




(39) 
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and the subscript g stands for Gaussian, as it represents the Gaussian contribution to the full 
joint probability. The remaining terms relate to approximations of the single node potentials 
in the following manner 



q{x) = qg{x)Y[T{x) 



(40) 



Here again qg{x) = ex.p[—{x — mg)^Tig ^{x — mg)/2] and f\x) is the standard Minka notation 
[9] for a term that approximates an intractable contribution, in our case 

fix) = fix.) = e-^'(^») (41) 

Updating the parameters m* and proceeds in the usual way: first, for a term i, the 
contribution of its approximation is removed from the full joint: 



ix) 



(x) 



meaning that 



iT.\')-^m\' = Sg ^mg + iJ:^)-^m^ 

Then the new value of the full parameters is obtained by defining 

q\'ix)f{xi) 



pix) 



and minimizing 



KLip\q) 



qix) 



J dxpix) log 

dxq\'ix)fix,)log 

dxq\'ix)fix,)\log 



f dxq\''{x)f{xi) 
Pix) 



j\Kx)fix,) 



qiix)q2ixi) 
fix.) 



q2ixi) 



+ log 



q\\x) 
Qlix) 



(42) 

(43) 
(44) 



(45) 



(46) 



where we have taken the liberty of decomposing the Gaussian function qix) into a Gaussian 
that depends only on x. and a remaining Gaussian depending on the whole vector x. Since 
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both qi{x) and q^'''{x) are Gaussians, the KL-divergence is minimal when they are equal and 
thus we have to minimize 



KL{p\q) = j dx,q\\x,)f\x,)\og^-^^ 

q2{xi) 



(47) 



with respect to q2{xi), where g^*(xj) = J dx\^^.q\^{x) and it is clear that q2{xi) is parameterized 
by (S*)^^ and m*, the only parameters to be updated. We furthermore deduce that these 
parameters contribute only to single entries in the matrices and vectors (i.e. they are scalars). 
Thus 



= Z ^ J dxi Xi q\\x.i)r{xi) 
= Z-' I dx, x\ q\\xi)f{xi) - (m^)2 
Z = I dx^ q\\xi)nxi) 



(48) 
(49) 
(50) 



The marginalization of q^'^ix) yields, 

g\*(xi) ~ exp 

where 



[xi -m- y 



\i 



2S 



1 



(51) 



m, 



g) + diag\i 



(52) 
(53) 



Thus the most costly computations are the inversion in equation (j52p and the one-dimensional 
integrations of ()50p . For very large models the inversions may become prohibitive, but other- 
wise this scheme seems efficient, since the "cavity covariances" do not have to be computed 
separately but are implicitly present in these inversions, and are optimal with respect to the 
minimization of the KL-divergence. 



6.2 Loop corrections formulation 

In this subsection we will discuss a possible generalization of the loop correction scheme for the 
model discussed in the previous subsection. The same model with nonlinear single-variable 
potentials may be tackled starting from the loop correction scheme at the beginning of this 
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paper, by slightly generalizing it to the case where il^iiai) — >■ ViCfi)^"^'^^'^- The formalism 
at the beginning of this paper may still be applied when the Gaussian parameterization of 
the distributions P^^^ctq.) for all i is an approximation. For given estimates of the covariance 
matrices Aj we then find: 



with 



m; 



J _ 



Vi + m,- 



/rfarxoxpj-c&ffa. m\ nf (.4,. {r;}))} 

/ da exp { -<^i {a, m% aj' {Ai, {vj}))} 

e] J dTT exp {-$;.(r, a;.(A„ {v\}))] 

/ dT exp { -$;.(r, mi, a5.(A, {i;,*}))} 

/ dgg^ exp m', c4(^,, {t;,'}))} 

/ da exp m% a|(Ai, {v\}))^ 

J dT{ml + e*r)2 exp |-$* (t, m^, a* (A, {^^;}))} 

/ drexp {-$;(t, mi, aj(A, {^1}))} 
/ daaexp{-^i{a,va.\ai{Ai,{vj}))} 

J da exp {-^i {a, m^ , (^j , { v[ } ) ) } 
/ daa'^ ey:p{-^i{a,m\ai{Ai,{vi}))} 

J da exp { (cr, m' , ( , {t;, }) ) } 

((j-m])2 



$j(a,m\ai(A,,W})) 



2«f 



+ Vi{a) 



l-SialiA,{vi}) 



m; 



J _ 



^i{a, in\ ai{Ai,{vi})) 



' k£di\j 
{a - rhif 



2vi 



+ Vi{a) 



Vi = 



rrii 



1 - SittiiAi, {v}}) 



(54) 



(55) 

(56) 
(57) 

(58) 
(59) 

(60) 

(61) 
(62) 

(63) 
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On the one hand, it is easy to check that these equations reduce to the BP equations with 
loop correction when Vi{ai) = for all i, i.e. rri'l = m'^ — e^jin^j and vj = vj — (ep^^'j- In that 
case they should be equivalent to the full Gaussian EP approach of the previous subsection 
as well, since both treatments are exact in this limit. On the other hand, when we take 
Vi{ai) 7^ and Ai = for all i, these updates are somewhat similar to EP with completely 
factorizing Gaussian approximate target distribution (i.e., deriving equations starting from 
diagonal S). The slight difference is due to the fact that the propagated expectations ml and 
vl parameterize approximate cavity distributions (i.e. in absence of one neighboring variable) 
and not the actual target marginal distributions. Thus the KL-divergence with an approximate 
factorizing cavity distribution is minimized and not with the approximate target distribution. 
When Vi{ai) = for all i, this difference vanishes, and the algorithm reduces to EP with a 
factorizing Gaussian as approximate joint distribution, which, in that limit (where integrals 
may be performed exactly) is equivalent to ordinary BP. 

When optimizing the approximations for marginal moments is the objective of the algo- 
rithm, the approach of this subsection is obviously not optimal, since instead moments of 
cavity distributions are optimized in the integrals that calculate the messages. 



6.3 Alternative loop correction formalism 

Inspired by the above observations regarding the optimization of the marginal moments of 
the target approximation, one may derive alternative consistency equations as in [7J, starting 
from the expressions for the actual marginals, such that the integrations include full sets of 
neighboring factors. Once again, we approximate the cavity distributions by Gaussians, and 
find 



with 



ml: 



(of, 



f daiajexp [-^i{ai,m\ai{Ai,{vl}))] 
J datexp [-^i{ai,m\ai{Ai,{vl}))] 

J dai exp [-<^i{ai,m\ai{Ai,{vl}))] 



(64) 
(65) 

(66) 
(67) 



For = this reduces to EP with fully factorizing Gaussian, and again Vi{ai) =0 leads to 
equations which are equivalent to BP. A suitable choice of Ai should make the fixed points of 
the above equations equivalent to the full-Gaussian EP equations at the beginning of this sec- 
tion, since both approaches optimize the marginal moments of each variable, given a Gaussian 
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interaction matrix with the rest of the model. However, the benefit of fuh Gaussian EP is that 
this Gaussian interaction matrix is optimized on the way, albeit at the cost of an inversion at 
each iteration, while the loop corrected approach desires an estimate of Ai as input, which is 
not further updated. 

Thus loop corrections are an alternative for the current type of model only if these inver- 
sions are so costly that approximations of the above form are sensible. 



6.4 Estimating Af. response propagation 



In the above formalism, an approximation for the cavity covariance matrix Ai may be obtained 
by applying a linear response algorithm to the graph from which variable i has been removed. 
The entries of Ai for variables j, k G di follow from 



(i) 



Sk- 



Sk 



dfJ-k 



(68) 
(69) 



Thus derivatives of average messages should be computed. Using ([57|) and we estimate 
them by taking the derivatives of these update equations neglecting the corresponding cavity 
covariances (setting Aj = for all j ^ i): 



dfJ-k 



(i) 
V ■ 
3 



djJ-k 



Sik , J dm^ 
^ 



(70) 
(71) 



The reasoning here is that an estimate of the covariance matrix in "zeroth order" enables a first 
order corrected version (see reference [5]) of the expectation propagation algorithm, equations 
()57p and (I63p with = Vi. Note that, given the values of the single node variances, which 
have to be obtained via running EP including integrations on the graph without variable 
i, we have a fast algorithm for the responses, that does not involve integrations. Thus the 
cost for a cavity covariance matrix is determined by the factorizing EP updates on the graph 
without the central variable i, after which the responses may quickly be obtained. This way 
of estimating Ai is obviously as costly as a number of inversions of matrices of dimension N 
In the full Gaussian EP approach, an inversion is necessary at each update, such that in the 
end the present approach might become cheaper, but for the current model EP itself seems 
more practicle. Possibly however, for more complex models the loop correction approach is 
beneficial, a topic which is to be further investigated. 
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7 Discussion 



In this paper we have derived loop corrected belief propagation equations for continuous 
variable models. In particular we have worked out the exactly tractable case of Gaussian 
models, and have derived the exact message passing equations. The role of the "connected 
correlations" of the cavity distribution discussed in [5j is taken over by the off-diagonal parts 
of the covariance matrix, which may be obtained in preprocessing algorithms. Moreover, 
using the fact that if ordinary BP converges, it produces exact marginal averages, various 
relations between BP messages are obtained, leading to alternative update schemes to invert 
the covariance matrix. 

For models involving nonlinear terms, for which approximation algorithms are needed in 
order to compute marginals, we discussed some relations between expectation propagation 
approaches and loop correction strategies, in particular for a model class where the nonlinear 
potentials involve only single variables. Loop correction approaches for continuous variable 
models become attractive once alternative strategies like expectation propagation grow too 
costly. On the other hand, however, they are themselves heavily penalized by the cost of the 
preprocessing stage necessary to estimate the cavity connected correlations. 
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A Full covariance matrix as a function of BP quantities 

Directly writing out the error a\''^ — af^ results in 



k,l&di 



(72) 



It follows that 



+ Jik{A. 

kedi\l 



m 



i kl 



(i) BP _i BP 
I 



m 



BP 



from which we may furthermore derive the relation 



{{Di+Ai)3i}- = J^jV 



r (i)BP i BPi 



m. 



BP 



(73) 



(74) 



A.l Off-diagonal parts of covariance matrix 

The off-diagonal elements of the covariance matrix of the model may be expressed in terms 
of cavity distributions again, using the formulation of subsection (|2.2p . We find, using the 
measures (flOl) . 



{aiaj) 



led. 



BP (i) BP 



JijVj 



ni] 



BP 



where we have used ()74p . 

Next nearest neighbor correlations follow from a similar calculation as above: 



(75) 



[CTjcrk) 



{(^j(^k)i 



(i) BP (i) BP , BP 
,BP\2 I „,LC 



j. A; € a, 

,f {(A + A,)Ji}^ + m« {(A + A,)J,} . 



+ [(mf^)^ + v^"^] {(A + ^i)^i}, {(A + Ai)Ji}^ + (A + 



(76) 



All terms follow from BP on the graph with and without i, except for the last one. Using 
()74p . and renaming terms, we obtain equations (1360 . 
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